fix(e2e): stop wp-env warm-up JSON parse flakes by mokagio · Pull Request #540 · wordpress-mobile/GutenbergKit

mokagio · 2026-07-06T06:49:59Z

I run into E2E failures from time to time in this repo over the past few weeks. The last time this happened, in the build for PR #533 which I was reviewing, I decided to task an agent to investigate with /goal E2E green consistently. This is the result.

The approach makes sense to me. While one should avoid polling in tests, I think it's acceptable in the context of E2E . But, I'm not familiar withe the Playground / wp-env system that's in use for the tests, so I can't really judge.

I'm sharing this in the hope it helps getting the E2E back to stable, even if just as a starting point.

AI-generated details below.

What?

Makes the Web E2E job resilient to wp-env warm-up hiccups, which were failing builds intermittently (e.g. build 2401).

Why?

Every failure in the flaky runs traced to wp-env-fixtures.js, where response.json() threw on a non-JSON body returned by the editor-settings/assets REST endpoints during warm-up — either a PHP notice printed into the response (<br /><b>Warning...) or a transient 500. Editor settings are cached in module scope, but only on success, so a single early hiccup poisoned the first fetch and failed every early spec while later ones passed — the classic flaky signature (7 failed / 41 passed with all failures in editor-load, editor-title, editor-error).

How?

Two complementary layers:

.wp-env.json: set WP_DEBUG_DISPLAY: false so PHP notices never leak into REST bodies. WP_DEBUG_LOG stays on, so they still land in debug.log.
e2e/wp-env-fixtures.js: route both fetches through a fetchJson helper that reads the body as text, retries transient 5xx/parse failures, and includes a body snippet in the final error instead of an opaque Unexpected token '<'.

Testing Instructions

This is a draft to exercise CI — the goal is a green (and repeatably green) Web E2E job.

Locally: make wp-env-start then make test-e2e. Before this change the settings-fetch failures reproduce on a cold instance; after, the suite passes (verified: 47/49 passed, the 2 remaining being local-only media-upload timeouts under fullyParallel workers, which pass at CI's workers: 1).

E2E runs failed intermittently when the wp-env REST endpoints returned a non-JSON body during warm-up — a PHP notice printed into the response (`<br /><b>Warning...`) or a transient `500` — and `response.json()` threw. Because editor settings are cached in module scope only on success, a single early hiccup poisoned the first fetch and failed every early spec while later ones passed: the flaky signature. Two complementary fixes: - Set `WP_DEBUG_DISPLAY: false` so PHP notices never print into REST bodies. `WP_DEBUG_LOG` stays on, so notices still land in `debug.log`. - Route the settings and assets fetches through a `fetchJson` helper that reads the body as text, retries transient `5xx`/parse failures, and surfaces a body snippet in the final error instead of an opaque `Unexpected token '<'`. --- Generated with the help of Claude Code, https://claude.com/claude-code Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wpmobilebot · 2026-07-06T06:55:58Z

XCFramework Build

This PR's XCFramework is available for testing. Add the following to your Package.swift:

.package(url: "https://github.com/wordpress-mobile/GutenbergKit", branch: "pr-build/540")

_{Built from 7f06856}

CI surfaced the real warm-up failure the earlier retry only papered over: the Playground runtime keeps installing plugins (Blueprint steps) after it starts serving requests. During those installs WordPress drops a `.maintenance` file and returns a `503`, and there is a race where the file is removed mid-request and PHP fatals — both corrupt the first editor REST call. Gate provisioning in `wp-env-setup.sh` on the settings endpoint returning valid JSON, so the suite never starts against a half-provisioned instance. Raise the in-test fetch retry budget (5s -> 30s) as a backstop for any mid-run blip. --- Generated with the help of Claude Code, https://claude.com/claude-code Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Improves stability of the Web E2E suite during wp-env warm-up by preventing PHP notices from corrupting REST responses and by making the fixtures/settings fetch more resilient to transient failures.

Changes:

Adds a fetchJson helper in e2e/wp-env-fixtures.js that retries transient HTTP/parse failures and surfaces a helpful body snippet on final failure.
Extends bin/wp-env-setup.sh to wait until the editor settings endpoint returns valid JSON before proceeding to run tests.
Updates .wp-env.json to disable WP_DEBUG_DISPLAY so PHP notices don’t leak into REST response bodies.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File	Description
`e2e/wp-env-fixtures.js`	Routes editor settings/assets fetches through a retrying JSON fetch helper to reduce warm-up flakes.
`bin/wp-env-setup.sh`	Adds a readiness gate for the editor settings REST endpoint to avoid early warm-up failures.
`.wp-env.json`	Disables debug display to keep REST responses clean while preserving debug logging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iangmaia · 2026-07-08T14:29:03Z

@claude review this PR.

claude · 2026-07-08T14:29:20Z

Claude finished @iangmaia's task in 0s —— View job

I'll analyze this and get back to you.

iangmaia · 2026-07-08T14:47:38Z

Claude finished @iangmaia's task in 0s —— View job

I'll analyze this and get back to you.

FYI #533 isn't merged so ofc this won't trigger the new Claude workflow.

mokagio · 2026-07-09T04:45:20Z

@claude review this PR, now that #533 landed and the workflow is secured.

claude · 2026-07-09T04:50:57Z

Claude finished @mokagio's task in 0s —— View job

I'll analyze this and get back to you.

mokagio · 2026-07-09T07:59:22Z

@iangmaia looks like the Claude review run but never went past reporting that it started? I had a quick look at the logs, https://github.com/wordpress-mobile/GutenbergKit/actions/runs/28994702797/job/86041799638, but couldn't make much of it. Haven't passed it to my AI for further investigation, but can do that later if you don't have the bandwidth.

mokagio self-assigned this Jul 6, 2026

github-actions Bot added the [Type] Bug An existing feature does not function as intended label Jul 6, 2026

This was referenced Jul 6, 2026

[Tooling] Secure Claude workflows #533

Merged

test(e2e): soak Web E2E ×10 (do not merge) #541

Closed

mokagio marked this pull request as ready for review July 6, 2026 15:30

Copilot AI review requested due to automatic review settings July 6, 2026 15:30

Copilot started reviewing on behalf of mokagio July 6, 2026 15:30 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

mokagio requested a review from jkmassel July 7, 2026 09:07

Merge branch 'trunk' into mokagio/fix-e2e-wp-env-flaky

7f06856

mokagio mentioned this pull request Jul 9, 2026

build(deps): Bump faraday from 1.10.5 to 1.10.6 #543

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(e2e): stop wp-env warm-up JSON parse flakes#540

fix(e2e): stop wp-env warm-up JSON parse flakes#540
mokagio wants to merge 3 commits into
trunkfrom
mokagio/fix-e2e-wp-env-flaky

mokagio commented Jul 6, 2026 •

edited

Loading

Uh oh!

wpmobilebot commented Jul 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

iangmaia commented Jul 8, 2026

Uh oh!

claude Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

iangmaia commented Jul 8, 2026 •

edited

Loading

Uh oh!

mokagio commented Jul 9, 2026

Uh oh!

claude Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

mokagio commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mokagio commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Testing Instructions

Uh oh!

wpmobilebot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

XCFramework Build

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

iangmaia commented Jul 8, 2026

Uh oh!

claude Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iangmaia commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mokagio commented Jul 9, 2026

Uh oh!

claude Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mokagio commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mokagio commented Jul 6, 2026 •

edited

Loading

wpmobilebot commented Jul 6, 2026 •

edited

Loading

claude Bot commented Jul 8, 2026 •

edited

Loading

iangmaia commented Jul 8, 2026 •

edited

Loading

claude Bot commented Jul 9, 2026 •

edited

Loading